Skip to content

fix(install): stop leaking temp files when an atomic write fails - #165

Open
rominf wants to merge 1 commit into
mainfrom
fix/atomic-downloads-no-partial-leaks
Open

fix(install): stop leaking temp files when an atomic write fails#165
rominf wants to merge 1 commit into
mainfrom
fix/atomic-downloads-no-partial-leaks

Conversation

@rominf

@rominf rominf commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

write_file_atomically leaves its scratch file behind when a write or rename
fails. The name embeds a millisecond timestamp, so each retry leaks a distinct
orphan instead of reusing one — on a full disk, retrying makes things worse.

  • Remove the temp file on every failure path, in both copies of the helper
    (apps/rocm/src/therock.rs, apps/rocmd/src/lib.rs).
  • Build the temp name from the whole file name, so a multi-extension artifact
    keeps its extensions: sdk.tar.gz yields sdk.tar.gz.tmp-<id> where
    with_extension dropped the .gz.

Root cause

Reproduced by filling a small filesystem and calling the helper twice:

attempt 0: ... No space left on device (os error 28)
  leftovers: [("artifact.tar.tmp-1785755921807", 67108864)]
attempt 1: ... No space left on device (os error 28)
  leftovers: [("artifact.tar.tmp-1785755921807", 67108864),
              ("artifact.tar.tmp-1785755921875", 0)]

Two attempts, two orphans, the first holding all the space that was left.

Scope

This PR originally also rewrote the download path. #198 has since landed
download_file_streaming, which covers that ground more thoroughly (sibling
.part file, resume, length and digest cross-checks), so those changes are
dropped in favour of it. What remains is the one defect #198 did not touch.

Note this does not close #158. Both in-call cleanup gaps are now fixed, but
a .part file orphaned by a hard crash or Ctrl-C still survives: the name is
keyed on the process id, so a later run never removes it, and nothing sweeps
for them. Leaving #158 open for that.

Test plan

  • write_file_atomically_cleans_up_temp_when_the_rename_fails (both crates) —
    portable, runs in CI. Fails before, passes after.
  • write_file_atomically_temp_name_preserves_multi_dot_file_names,
    temp_sibling_path_preserves_multi_dot_file_names.
  • write_file_atomically_cleans_up_temp_on_write_failure (both crates) —
    the original /dev/shm ENOSPC reproduction, #[ignore]d because it fills a
    shared tmpfs. Run with cargo test -- --ignored.

Risk: low. Two self-contained helpers; behaviour on the success path is
unchanged apart from the temp file's name.

Refs #158

juhovainio
juhovainio previously approved these changes Aug 6, 2026

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@rominf
rominf added this pull request to the merge queue Aug 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 6, 2026
@rominf
rominf force-pushed the fix/atomic-downloads-no-partial-leaks branch from 8c1ec0c to 7204154 Compare August 7, 2026 07:53
@rominf

rominf commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Merge-queue ejection: root cause and fix

This PR was ejected from the merge queue when the blocking E2E tests job failed on dash-managed-service-metrics (merge-group run 31102672143, job 92621417125):

FAIL: 'dash-managed-service-metrics' was expected to pass on this host but FAILED — a regression.
Step panicked: TTFT metrics did not appear: timed out after 30s waiting for "50ms".

Root cause

The captured screen in that log shows the dashboard still on Home — the 4 keypress that should have switched to Observe never took effect, so the step waited 30s for metrics on a view it never reached.

The two scenarios that open Observe launch the TUI and send the tab key with no assertion in between, unlike the demo-data journeys which assert the home view first. A key written into the pseudo-terminal before the dashboard is reading input can be consumed by whatever holds the terminal at that moment, and nothing retries it. The step then fails much later, in an assertion about a view the dashboard never left — which is exactly the failure signature above.

This is a latent synchronisation defect in the E2E step, not in this PR's production code: nothing in the download/atomic-write change is reachable from rocm dash. It surfaced here rather than in this PR's own checks because those ran against the older base.

Fix

Send the tab key repeatedly until the Observe chip is actually marked active, so the step depends on the dashboard having acted on the key rather than on it having been ready when the key was written. The helper is restricted to idempotent keys (a tab jump, not a toggle).

Reproduction

Deterministic, by pointing the harness at a wrapper that drains the terminal before exec'ing the real binary — i.e. a dashboard that is not yet reading when the key arrives:

cat > /tmp/wrapper/rocm <<'SH'
#!/bin/sh
stty raw -echo min 0 time 20; dd bs=1 count=16 of=/dev/null 2>/dev/null; stty sane
exec /path/to/target/release/rocm "$@"
SH
ROCM_CLI_BINARY=/tmp/wrapper/rocm cargo xtask e2e -- -n "Observe displays metrics"

Before: 1 scenario (1 failed) with the CI symptom verbatim. After: 1 scenario (1 passed). The sibling scenario (dash-loading-service-status), which uses the same step, also passes under the wrapper.

Verified in addition: full cargo xtask e2e matches the base commit exactly (the two diagnose-* failures are pre-existing on this host, confirmed on ec2bcb32 itself), cargo fmt --all --check, both clippy invocations, cargo xtask manifest --check, and cargo test --workspace --all-targets (only two pre-existing proc_lifecycle failures local to this host).

Note: the guard is exercised by these two scenarios on every run, but the key-loss condition itself is only reproducible with the wrapper above, so there is no automated test that would fail without it.

@rominf
rominf force-pushed the fix/atomic-downloads-no-partial-leaks branch from 7204154 to 78851e6 Compare August 11, 2026 16:14
write_file_atomically named its scratch file path.tmp-<unix_ms> and
returned early on a write or rename error without removing it. Because
the name embeds a timestamp, each retry left a distinct orphan rather
than reusing one — so on a full disk, retrying made things worse: the
first orphan held all the space that remained.

Remove the temp file on every failure path in both copies of the helper,
and build the name from the whole file name so a multi-extension
artifact keeps its extensions: sdk.tar.gz now yields
sdk.tar.gz.tmp-<id>, where with_extension dropped the .gz.

Refs #158

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/atomic-downloads-no-partial-leaks branch from 78851e6 to a1a7ad1 Compare August 14, 2026 12:13
@rominf rominf changed the title fix(download): never leave partial files behind fix(install): stop leaking temp files when an atomic write fails Aug 14, 2026
@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Re-scoped — please re-review, the existing approval predates this

Heads up that this PR now does substantially less than when it was approved. The
approval at 8c1ec0c was for code that is largely no longer here, and this repo
does not dismiss stale reviews on push, so I have re-requested review rather than
let it merge on a review of code that no longer exists.

Dropped: the download half. #198 (c6b5bb5) landed download_file_streaming
on main, which implements the same idea more completely than this PR did —
sibling .part file, rename on success, plus resume via Range, Content-Range
offset validation, SHA-256 and Content-Length cross-checks, a free-space
preflight, and retry classification. Keeping this PR's version would have meant a
second, weaker download path, so it is gone in favour of main's.

Kept: the write_file_atomically temp leak, which #198 did not touch. Both
copies (apps/rocm/src/therock.rs, apps/rocmd/src/lib.rs) still returned early
on a write or rename failure without removing the scratch file, and the name
embeds a timestamp so each retry leaked a distinct orphan. #198 stopped routing
the SDK tarball through the therock.rs copy, but three callers remain, so both
copies are fixed here.

Split out: the e2e TUI flake fix. The Observe-tab keystroke retry was
unrelated scope creep and is moving to its own PR.

Fixes #158 is now Refs #158. The in-call cleanup gaps are closed, but the
issue's actual complaint — orphans that are never cleaned up — survives. A .part
file left by a hard crash or Ctrl-C is keyed on the process id, so the next run's
cleanup targets a different path and never removes it, and nothing in the tree
sweeps for them. #158 should stay open; I have not touched it.

@rominf
rominf requested a review from a team August 14, 2026 12:14
@rominf
rominf dismissed juhovainio’s stale review August 14, 2026 12:26

Dismissing automatically: this PR has been re-scoped since this approval. The download rewrite that was reviewed here has been dropped, because #198 (c6b5bb5) landed a more complete implementation of the same idea on main. What remains is a different change — the write_file_atomically temp leak in both copies of the helper — plus a corrected Refs #158 (it does not close #158; a .part file orphaned by a hard crash still survives, since the name is keyed on the pid). Re-review requested. See the re-scope comment for detail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Issue]: Interrupted downloads leave orphaned partial files that are never cleaned up

2 participants